In the prior art, it has been well known that computer systems can be used to index records of a database. In recent years, a unique distributed database has emerged in the form of the World-Wide-Web (Web). The database records of the Web are in the form of pages accessible via the Internet. Here, tens of millions of pages are accessible by anyone having a communications link to the Internet.
The pages are dispersed over millions of different computer systems all over the world. Users of the Internet constantly desire to locate specific pages containing information of interest. The pages can be expressed in any number of different character sets such as English, French, German, Spanish, Cyrillic, Kanakata, and Mandarin. In addition, the pages can include specialized components, such as embedded "forms," executable programs, JAVA applets, and hypertext.
Moreover, the pages can be constructed using various formatting conventions, for example, ASCII text, Postscript files, html files, and Acrobat files. The pages can include links to multimedia information content other than text, such as audio, graphics, and moving pictures. As a complexity, the Web can be characterized as an unpredictable random update, insert, and delete database with a constantly changing morphology.
One characteristic of the World-Wide-Web makes it relatively easy to copy Web pages from one site to another. Web users frequently incorporate pages created by others into their own pages to streamline access. It is estimated that as much as 25% of the Web is composed of duplicate pages. If all the duplicate pages are fully indexed, the amount of storage required for the index would greatly increase.
Therefore, it is desired to provide a technique which minimizers the likelihood that duplicate pages are indexed The technique should also allow for reindexing as duplicate pages are deleted.